Association processes and related systems for manipulators

ABSTRACT

A computer-assisted medical system includes manipulators, a user input system, a user output system comprising a display device, and a controller configured to execute instructions to perform operations. The operations include, in a pairing mode and in response to a first set of signals generated by the user input system, causing a virtual selector shown on the display device to move relative to an imagery shown on the display device. The operations further include, in the pairing mode, associating a first manipulator with a portion of the user input system based on movement of the virtual selector relative to a represented location of the first instrument, and, in a following mode, controlling motion of the first instrument in accordance to a second set of signals generated by the user input system in response to operation of the portion of the user input system by a user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional of and claims priority to U.S.Provisional Patent Application No. 62/537,795, filed on Jul. 27, 2017and U.S. Provisional Patent Application No. 62/551,702, filed on Aug.29, 2017. The entire contents of each of the foregoing applications arehereby incorporated by reference.

TECHNICAL FIELD

This specification relates to association processes and related systemsfor manipulators, for example, for teleoperated manipulators.

BACKGROUND

Robotic manipulators can be operated to control motion of instruments ina workspace. For example, such manipulators can be used to performnon-medical and medical procedures. As a specific example, teleoperatedsurgical manipulators can be used to perform minimally invasive surgicalprocedures. An operator can control the manipulators using a usercontrol system, e.g., connected wirelessly or via a wired connection tothe teleoperated manipulators. The user control system can includemultiple user input devices such that each of the teleoperatedmanipulators can be controlled by a distinct user input device of theuser control system. The operator can thus independently control each ofthe teleoperated manipulators using the user input devices.

SUMMARY

In one aspect, a computer-assisted medical system includes teleoperatedmanipulators, a user input system, a user output system comprising adisplay device, and a controller configured to execute instructions toperform operations. The operations include, in a pairing mode and inresponse to a first set of signals generated by the user input system,causing a virtual selector shown on the display device to move relativeto an imagery shown on the display device. The imagery represents alocation of a first instrument supported by a first manipulator of theplurality of manipulators and a location of a second instrumentsupported by a second manipulator of the plurality of manipulators. Theoperations further include, in the pairing mode, associating the firstmanipulator with a portion of the user input system based on movement ofthe virtual selector relative to the represented location of the firstinstrument, and, in a following mode, controlling motion of the firstinstrument in accordance to a second set of signals generated by theuser input system in response to operation of the portion of the userinput system by a user.

In another aspect, a method of operating a computer-assisted medicalsystem including a plurality of teleoperated manipulators is featured.The method includes causing a display device to present imageryrepresenting a location of a first instrument supported by a firstmanipulator of the plurality of manipulators and a location of a secondinstrument supported by a second manipulator of the plurality ofmanipulators, and a virtual selector movable relative to the imagery inresponse to a first set of signals generated by a user input system. Themethod further includes associating, in a pairing mode, the firstmanipulator with a portion of the user input system based on movement ofthe virtual selector relative to the represented location of the firstinstrument, and, controlling, in a following mode, motion of the firstinstrument in accordance to a second set of signals generated by theuser input system in response to operation of the portion of the userinput system by a user.

In yet another aspect, one or more non-transitory computer readablemedia is featured. The one or more non-transitory computer readablemedia store instructions that are executable by a processing device andupon such execution cause the processing device to perform operations.The operations include causing a display device to present, imageryrepresenting a location of a first instrument supported by a firstmanipulator of a plurality of teleoperated manipulators and a locationof a second instrument supported by a second manipulator of theplurality of manipulators, and a virtual selector movable relative tothe imagery in response to a first set of signals generated by a userinput system. The operations further include associating, in a pairingmode, the first manipulator with a portion of the user input systembased on movement of the virtual selector relative to the representedlocation of the first instrument, and, controlling, in a following mode,motion of the first instrument in accordance to a second set of signalsgenerated by the user input system in response to operation of theportion of the user input system by a user.

Advantages of the foregoing may include, but are not limited to, thosedescribed below and herein elsewhere. For example, associations betweenuser-operable portions of a user input system and teleoperatedmanipulators can be formed in a manner that is intuitive for theoperator. Rather than having to interact with lists and informationpresented on a display that do not provide the operator with a sense ofconfigurations or relative poses of the teleoperated manipulators, anoperator can control a virtual selector overlaid on or otherwiseoverlapping with imagery of a workspace to initiate association betweena user-operable portion and a particular teleoperated manipulator. Inparticular, by controlling a location of a virtual selector relative toimagery representative of a workspace of instruments supported by themanipulators, the operator can intuitively operate the user input systemto associate the user-operable portion and the teleoperated manipulator.

Human-detectable feedback can be provided during the pairing mode sothat the operator can be kept apprised of states and processes ofdevices, e.g., the user-operable portions of the user input system andthe manipulators to be associated. For example, the controller cangenerate feedback indicative of association states of user-operableportions, the manipulators, or both the user operable portions and themanipulators. Based on the feedback, the operator can initiateassociation processes for devices that have not already been associated.In addition, the controller can generate feedback indicative of aproposed association prior to finalizing an association between auser-operable portion and a manipulator. This enables the operator tomake adjustments to a proposed association, thereby providing theoperator with greater control during the association process. In someimplementations, human-detectable feedback can be continued or newlyprovided after an association has been made, and indicate the portion ofthe user input system that is associated with a particular manipulator,and vice versa. Further, the controller can disassociate a user inputdevice or a manipulator in response to user input or a system event.

Although some of the examples described herein often refer to medicalprocedures and medical instruments, the techniques disclosed also applyto non-medical procedures and non-medical instruments. For example, theinstruments, systems, and methods described herein may be used fornon-medical purposes including industrial uses, general robotic uses,manipulation of non-tissue work pieces, and/or cosmetic improvements.Other non-surgical applications include use on tissue removed from humanor animal anatomies (without return to a human or animal anatomy) or onhuman or animal cadavers.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other potential features, aspects,and advantages will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a top view of a system including a manipulator and a console.

FIG. 1B is a front view of the console of FIG. 1A.

FIG. 1C is a top view of non-console based user input and user outputsystem that can replace the console shown in 1A and 1B.

FIG. 2 is a front perspective view of a manipulator and a patient on anoperating table.

FIGS. 3A and 3B are views of a display device during a process toassociate user-operable portions of a user input system withmanipulators.

FIG. 4 is a block diagram of a system for performing a manipulatorassociation process.

FIG. 5 illustrates associations between manipulators and user-operableportions of a user input system.

FIG. 6 is a flowchart illustrating a process of operating a user inputsystem to control a manipulator.

FIG. 7A is a flowchart illustrating a process to associate auser-operable portion with a manipulator.

FIGS. 7B-7E are views of a display device during a process to associateuser-operable portions of a user input system with manipulators.

FIG. 8A is a flowchart illustrating a process to optimize associationsformed between manipulators and user-operable portions of a user inputsystem.

FIG. 8B illustrates, on a left side, a user input system and a displaydevice showing instruments and, on a right side, a top view ofmanipulators supporting the instruments.

FIG. 9 is a flowchart illustrating a process to reorient user-operableportions of a user input system.

FIG. 10 is a schematic diagram of a computer system.

FIG. 11 is a front view of a manipulator system.

FIG. 12 is a top view of a system including a manipulator and a motiondetection system.

FIG. 13 is a front view of a console including an eye tracking system.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION Example Systems

Referring to FIG. 1A, a system 100 in an environment 10 includes amanipulator system 101 including teleoperated manipulators 102 a, 102 b,102 c, 102 d (collectively referred to as manipulators 102 orteleoperated manipulators 102). The manipulators 102 are termed“teleoperated manipulators” because they that can be teleoperated by anoperator 104 through a physically separate user input system 106. Insome implementations, the manipulators 102 can also be controlleddirectly through manual interaction with the manipulators 102themselves. Thus, “teleoperated manipulators” as used in thisapplication include manipulators that can be controlled only throughteleoperation, and manipulators that can be controlled throughteleoperation and through direct manual control. The manipulators 102include movable portions that can support instruments (not shown), e.g.,surgical and medical instruments. The movable portions, for example,correspond to distal ends 112 a, 112 b, 112 c, 112 d of the manipulators102.

Referring also to FIGS. 1B and 1C, FIG. 1B is a front view of theconsole of FIG. 1A. FIG. 1C is a top view of non-console based userinput and user output system that can replace the console shown in 1Aand 1B. The operator 104 can teleoperate the manipulators 102 andmonitor instruments supported by the manipulators 102 using a user inputsystem and a user output system, e.g., including a display device. Insome examples, such as shown in FIGS. 1A and 1B, a standalone console103 b includes the user input system 106 b and the user output system.In the example shown, the console 103 includes user input systemportions such as input devices 108 a, 108 b, and a user output systemcomprising a stereoscopic display device 107 b).

In some examples, such as shown in FIG. 1C, a non-console based userinput and output system 113 may be used. In the example shown in FIG.1C, the user input and output system 113 includes a user input system106 c comprising handheld user input devices 108 c, 108 d. The handhelduser input device 108 c, 108 d are portions of the user input system 106c whose motion is not physically constrained by links and joints to abase or console. The user input system 106 c further includes a sensorsystem 109 that communicates with the user input devices 108 c, 108 dfor detecting user input. The user input and output system 113 furtherincludes a user output system including a monitor-type display device107 c that provides monoscopic or 3D images in various implementations.For convenience of explanation below, 106 is used to refer to user inputsystems generally and 106 b, 106 c are used to refer to the specificexamples shown in FIGS. 1B, 1C. Similarly, 107 is used to refer todisplay devices comprising user output systems generally, while 107 b,107 c are used to refer to the specific examples shown in FIGS. 1B, 1C.

When the system 100 is operated in a following mode, the operator 104can operate the user input system to generate a set of user inputsignals to control motion of the manipulators 102. FIGS. 1B and 1C showscontroller 110 as physically located with the user input and outputsystems. However, controller 110 may be physically separate from theuser input and output systems and communicate with the user input andoutput systems via electronic signals transmitted via wired or wirelesstechnology.

During operation of the system 100, the operator 104 can view thedisplay device 107 to view imagery, e.g., two-dimensional imagery orthree-dimensional imagery, representing the instruments mounted on themanipulators 102 while the manipulators 102 are being controlled by theoperator 104. For example, an instrument including an image capturedevice such as a camera is mounted to one of the manipulators 102. Theimage capture device generates imagery of distal end portions of otherinstruments mounted to the other manipulators 102. During operation ofthe system 100, the operator 104 can monitor poses of distal endportions of the instruments using the imagery presented on the displaydevice 107.

The user input system 106 is connected to the manipulators 102, e.g.,wirelessly or using a wired connected. The user input system 106includes multiple distinct portions operable by the operator 104 tocontrol operations of the manipulators 102. These user-operableportions, in some cases, correspond to distinct user input devices. Inthe example depicted in FIGS. 1B and 1C, the user input system 106includes manually operable user input devices (e.g., 108 a, 108 b areshown for 106 b, and 108 c, 108 d are shown for 106 c) (collectivelyreferred to as user input devices 108) corresponding to theuser-operable portions of the user input system 106 and movable relativeto the manipulators 102 to control movement of the manipulators 102. Theuser input system 106 can include other user input devices, e.g.,keyboards, touchscreens, buttons, foot pedals, etc., in addition to theuser-operable portions operated to control movement of the manipulators102 or other operations of the manipulators 102 in the following mode.These other user-operable portions can be used to allow user control ofthe display device 107 and otherwise allow user control of otheroperations of the system 100.

As described herein, in response to operation of a user-operable portionin a pairing mode, a controller 110 (shown in FIGS. 1B and 1C) of thesystem 100 can associate the user-operable portion of the user inputsystem 106 with a corresponding one of the manipulators 102. Whenassociated, the user-operable portion can be operated to control thecorresponding manipulator in a following mode to perform an operation,e.g., a medical operation, a surgical operation, a diagnostic operation,etc.

FIG. 2 shows an example of the manipulator system 101. For simplicity,only the manipulators 102 a, 102 b of the manipulator system 101 areshown. In some implementations, the manipulator system 101 includes asingle manipulators or includes three or more manipulators, e.g., fourmanipulators 102 a, 102 b, 102 c, 102 d as depicted in FIG. 1A.

Although FIG. 2 is described with respect to the manipulators 102 a, 102b, the manipulators 102 c, 102 d of FIG. 1A can include features similarto those presented with respect to the manipulators 102 a, 102 b. Themanipulators 102 a, 102 b, 102 c, 102 d may differ from one another inthat different instruments may be mounted to the manipulators 102 a, 102b, 102 c, 102 d. In addition, the manipulators 102 a, 102 b, 102 c, 102may be supported by an operating table 105 at different locations alongthe operating table 105.

The manipulators 102 a, 102 b include portions movable about a workspace114. For example, these portions can correspond to distal ends 112 a,112 b of the manipulators 102 a, 102 b that are movable about theworkspace 114. The distal ends 112 a, 112 b support instruments 116 a,116 b such that the instruments 116 a, 116 b can be moved about theworkspace 114 when the distal ends 112 a, 112 b are moved about theworkspace 114. In some implementations, actuation modules 117 a, 117 bare supportable at the distal ends 112 a, 112 b of the manipulators 102a, 102 b. The actuation modules 117 a, 117 b are removably mounted tothe distal ends 112 a, 112 b of the manipulators 102 a, 102 b andinclude one or more actuators that are operable to generate insertionand roll motions of the instruments 116 a, 116 b. The instruments 116 a,116 b are insertable through the actuation modules 117 a, 117 b suchthat the instruments 116 a, 116 b are attached to the actuation modules117 a, 117 b, which in turn are attached to the distal ends 112 a, 112 bof the manipulators 102 a, 102 b.

The manipulators 102 a, 102 b include powered joints 118 a, 118 b thatcan be driven to move the distal ends 112 a, 112 b of the manipulators102 a, 102 b about the workspace 114. Each of the manipulators 102 a,102 b includes multiple powered joints 118 a, 118 b that enable motionof the distal ends 112 a, 112 b in multiple degrees of freedom, e.g.,pitch, yaw, and roll motions of the distal ends 112 a, 112 b of themanipulators 102 a, 102 b. The instruments and manipulators describedherein can have one or more degrees of freedom that vary inimplementations. For example, the one or more degrees of freedom includeone or more of a yaw motion of the distal portion of the manipulator, apitch motion of the distal portion of the manipulator, an insertionmotion of the instrument supported by the manipulator, a roll motion ofthe instrument, a yaw motion of the end effector of the instrument, awrist motion of an end effector of the instrument, or a jaw or gripmotion of the end effector of the instrument.

The system 100 is a computer-assisted system. For example, thecontroller 110 can control operation of the system 100 and coordinateoperations of the various subsystems of the system 100, including butnot limited to the manipulators 102, the user input system 106, and theuser output system. While schematically depicted as a controller of theconsole 103, in some implementations, the controller 110 can include oneor more processors external to the console 103 and can be operable tocontrol any subsystem of the system 100, e.g., the manipulators 102 andthe console 103.

In some examples, the controller 110 can control operation of theactuators of the powered joints 118 a, 118 b as well as actuators of theactuation modules 117 a, 117 b. The distal ends 112 a, 112 b of themanipulators 102 a, 102 b, and hence the instruments 116 a, 116 b, aremovable about the workspace 114 when the user-operable portions of theuser input system 106 associated with the manipulators 102 a, 102 b areoperated by the operator 104.

In a following mode, a follower of a manipulator moves in response tomovement of a leader. The movement of the follower can emulate themovement of the leader. For a particular manipulator for example, theleader can be one or more of the user input devices 108, and thefollower can be one or more components of the manipulator. The followercan be an end effector of the manipulator, a remote center of themanipulator, or some other component of the manipulator. In someexamples, in the following mode, the distal ends 112 a, 112 b are thefollowers. For example, actuators of the powered joints 118 a, 118 b canbe controlled to generate motion of links of the manipulators 102 a, 102b about the powered joints 118 a, 118 b, thereby repositioning thedistal ends 112 a, 112 b of the manipulators 102 a, 102 b. The motionsof the distal ends 112 a, 112 b emulate the motions of the user inputdevices 108. In other examples, the motion of the user input devices 108in the following mode can cause an instrument mounted to the distal end112 a or 112 b to be ejected from the distal end 112 a or 112 b. Infurther examples, in the following mode, the actuation modules 117 a,117 b can be controlled to generate insertion motion of the instruments116 a, 116 b or to actuate the end effectors of the instruments 116 a,116 b.

Referring to FIGS. 1A, 1B, 1C, and 2, in some implementations, thesystem 100 is a medical system to perform a medical procedure on apatient 120. For example, the system 100 is a diagnostic system that canbe used to perform diagnostics on the patient 120. Alternatively oradditionally, the system 100 is a surgical system that can be used toperform a surgical operation on the patient 120.

A variety of alternative computer-assisted teleoperated instruments 116a, 116 b can be used. For example, the teleoperated instruments 116 a,116 b can be surgical instruments of different types having differingend effectors. In some cases, the instruments 116 a, 116 b includemultiple DOFs such as, but not limited to, roll, pitch, yaw, insertiondepth, opening/closing of jaws, actuation of staple delivery, activationof electro-cautery, and the like. Motion in at least some of such DOFscan be generated by the actuation modules 117 a, 117 b of themanipulators 102 a, 102 b to which the instruments 116 a, 116 b areselectively coupled.

If the instruments 116 a, 116 b are medical or surgical instruments,possible end effectors include, for example, DeBakey Forceps,microforceps, and Potts scissors include first and second end effectorelements that pivot relative to each other so as to define a pair of endeffector jaws. Other end effectors, including scalpels andelectrocautery probes, have a single end effector element. Forinstruments having end effector jaws, the jaws will often be actuated bysqueezing the grip members of input devices. Instruments can includeflexible shafts, the shafts being deflectable to enable repositioning ofthe distal ends of the shafts. In some cases, one or more of theinstruments 116 a, 116 b includes an image capture device. Examples ofinstruments with image capture devices include endoscopes, ultrasonicprobes, fluoroscopic probes, etc. The image capture device can captureimagery of other instruments in the workspace 114 (shown in FIG. 2), andthis imagery can be presented to the operator 104 to allow the operator104 to visually monitor positions of other instruments in the workspace114. Paired user-operable portions of the user input system 106 can alsobe used to control actuation of the end effectors.

FIGS. 3A and 3B show an example of the display device 107. Theseexamples depict the display device 107 in a pairing mode in which thedisplay device 107 presents representations 122 a, 122 b of theinstruments 116 a, 116 b and presents a virtual selector 121. In thepairing mode, the operator 104 operates the user input system 106 toreposition, e.g., to translate or to rotate, the virtual selector 121 toassociate the manipulators 102 a, 102 b supporting the instruments 116a, 116 b with corresponding user-operable portions of the user inputsystem 106, e.g., the user input devices 108. While FIGS. 3A, 3B, and 4are described with respect to association of the user input devices 108,in other implementations described herein, other examples ofuser-operable portions of a user input system can be associated with themanipulators 102 a, 102 b.

Turning to the example of FIGS. 3A and 3B, the imagery presented on thedisplay device 107 is captured by one or more image capture devices. Insome examples, as described herein, an image capture device is coupledto an instrument mounted to one of the manipulators 102 to captureimagery of instruments, e.g., the instruments 116 a, 116 b, coupled tothe other manipulators 102, e.g., the manipulators 102 a, 102 b. In someexamples, the image capture device is a stationary image capture devicein the environment 10 that captures imagery of the instruments 116 a,116 b.

In the example shown in FIG. 3A, the display device 107 presents imageryincluding the representations 122 a, 122 b of the instruments 116 a, 116b. The representations 122 a, 122 b represent a location of theinstrument 116 a supported by the manipulator 102 a and a location ofthe instrument 116 b supported by the manipulator 102 b, respectively.

The representations 122 a, 122 b can be part of imagery captured by animaging system, e.g., an endoscope. The captured imagery as presented onthe display device 107 is unchanged from when it is captured by theimaging system. In other examples, the representations 122 a, 122 b arepart of imagery captured by an imaging system but then is altered insome manner. For example, the imagery can be altered to includehighlighting, edge-finding, overlaid text, overlaid graphics, or otherindicators. In further examples, the representations 122 a, 122 b arepart of a synthetic image constructed in part or wholly from sensorinformation, e.g., collected by a sensor system 200 described herein andshown in FIG. 4. For example, the sensor information can includeinformation collected from a shape sensing sensor or other kinematicinformation for joints and links of the manipulators 102 a, 102 b.

In implementations in which the imagery is two-dimensional imagery, arepresented location of one of the instruments 116 a, 116 b in theimagery can be a single point in the imagery, a set of points in theimagery, or a two-dimensional region in the imagery. In implementationsin which the imagery is three-dimensional imagery, a representedlocation of one of the instruments 116 a, 116 b can be a single point inthe imagery, a set of points in the imagery, a two-dimensional region inthe imagery, or a three-dimensional volume in the imagery. Forthree-dimensional imagery, two or more image capture devices or an imagecapture device with depth sensing or stereoscopic configuration may beused to capture imagery to form the representations shown on the displaydevice 107.

The representations 122 a, 122 b of the instruments 116 a, 116 b in theimagery are indicative of relative poses, e.g., positions, orientations,or both, of the instruments 116 a, 116 b in the workspace 114 (shown inFIG. 2) of the instruments 116 a, 116 b. In some examples, the imageryis digital imagery captured by an image capture device coupled toanother instrument in the workspace, and the representations 122 a, 122b thus correspond to portions of the captured digital imagery.Alternatively or additionally, the imagery is a rendering generatedbased on imagery captured by the image capture device. In such cases,the representations 122 a, 122 b correspond to graphic indicators orportions of the rendering indicative of relative poses of theinstruments 116 a, 116 b.

The virtual selector 121 is a graphic indicator. The virtual selector121 has a roughly arrow-head, triangular shape in FIG. 3A; in otherimplementations, the virtual selector 121 may have any appropriateshape, including symmetric shapes such as circles. In someimplementations, the virtual selector 121 is a two-dimensional virtualselector overlaid on two-dimensional or three-dimensional imagerypresented by the display device 107. In some implementations, thevirtual selector 121 is a three-dimensional virtual selector overlaid onthree-dimensional imagery presented by the display device 107. In someimplementations, the virtual selector 121 represents a two-dimensionalor three-dimensional rigid body. In some implementations, the virtualselector 121 represents a compliant body or an assembly of componentscoupled with one or more joints allowing internal degrees of freedom. Insome implementations, in addition to, or instead of, being defined byits geometry, the virtual selector 121 is further defined by acoloration, a pattern, a blinking light, or other graphical property.Similar to the geometry of the virtual selector, this graphical propertycan be symmetric or asymmetric. In other implementations, the virtualselector is an augmented reality element. For example, the virtualselector can be a graphic indicator overlaid on a portion of theenvironment.

A location of the virtual selector 121 (including changes in locationthrough motion of the virtual selector 121), orientation, or acombination of location and orientation is controllable by the operator104 through operation of the user input system 106. The virtual selector121 has multiple degrees of freedom of motion relative to the imagerypresented by the display device 107. The degrees of freedom for thevirtual selector 121 allows the virtual selector 121 to be movable inthe space (e.g., two-dimensional space or three-dimensional space)represented by the imagery. In some implementations, the virtualselector 121 includes fewer than six degrees of freedom, e.g., fivedegrees of freedom without roll in three-dimensional space, threedegrees of freedom (translation and rotation) in two-dimensional space,two degrees of freedom without rotation in two-dimensional space, etc.

When the imagery and the representations 122 a, 122 b arethree-dimensional representations, the virtual selector 121 can betranslatable and rotatable in a three-dimensional space represented inthe imagery. In some examples, the virtual selector 121 has six degreesof freedom, including three translational degrees of freedom (e.g.,horizontal movement along a first axis, horizontal movement along asecond axis, and vertical movement) and three rotational degrees offreedom (e.g., yaw, pitch, roll).

In some implementations, the imagery presented on the display device 107can correspond to a two-dimensional projection of the workspace(including the instruments 116 a, 116 b), the projection being formedbased on imagery captured by a two-dimensional digital imagery capturedevice. The imagery presented on the display device 107 represents theinstruments 116 a, 116 b in two-dimensional space (e.g., therepresentations 122 a, 122 b are two-dimensional representations of theinstruments 116 a, 116 b). The degrees of freedom of the virtualselector 121 can include two translational degrees of freedom and tworotational degrees of freedom.

The controller 110 is configured to, in the pairing mode, operate thedisplay device 107 to present the virtual selector 121 for userselection of a manipulator, e.g., one of the manipulators 102, to beassociated with a particular user input device. The controller 110 mayoperate the display device 107 in any appropriate matter. For example,the controller may directly drive the display device 107 and providepixel-by-pixel instructions for rendering images. As another example,the controller 110 may provide one or more images for a displaycontroller of the display device 107 to render, to blend, or to blendand render. As a further example, the controller 110 may providedirections to a coprocessor or a display processing system to determinethe images to be displayed, and the coprocessor or display processingsystem would operate the display device 107 to display such images.

The virtual selector 121 is repositionable and/or reorientable inresponse to operation of the user input system 106 by the operator 104to form an association between a particular manipulator and a particularuser input device. In particular, the operator 104 controls the locationand/or orientation of the virtual selector 121 relative to therepresentations 122 a, 122 b of the instruments 116 a, 116 b to selectone of the manipulators 102 for association.

The controller 110 generates a control signal to cause the displaydevice 107 to reposition the virtual selector 121 in response to a userinput signal generated by the user input system 106 when the operator104 operates the user input system 106. The user input signal can begenerated based on an operator intent to move the virtual selector 121in one or more degrees of freedom. In some implementations, thecontroller 110 receives a user input signal indicative of movement inmultiple degrees of freedom and generates a control signal to cause thevirtual selector 121 to move along a subset of the multiple degrees offreedom. For example, repositioning of the virtual selector 121 in oneor more of the multiple degrees of freedom may not be visible to theoperator 104. If the imagery is two-dimensional imagery presented on thedisplay device 107 and the display device 107 only presentstwo-dimensional imagery, the subset of the multiple degrees of freedomcan include a horizontal translation degree of freedom and a verticaltranslation degree of freedom. This subset of the multiple degrees offreedom excludes another horizontal translation degree of freedom inwhich motion of the virtual selector 121 would not be represented on thedisplay device 107.

In some cases, the virtual selector 121 is axisymmetric, e.g., withrespect to geometry, graphical properties, or both. For example, thevirtual selector 121 is a cone, an arrow, a prism, or other axisymmetricshape. In some implementations, the virtual selector 121 is axisymmetricabout one, two, or more axes. By being axisymmetric, the virtualselector 121 does not appear to be repositioned due to rotation about aparticular axis. In this regard, the subset of the multiple degrees offreedom can include one or two of three available rotational degrees offreedom. This subset of the multiple degrees of freedom excludes arotational degree of freedom about the axis about which the virtualselector 121 is axisymmetric.

FIGS. 3A and 3B illustrate a process of associating the instruments 116a, 116 b or their corresponding manipulators 102 a, 102 b (shown in FIG.2) to the user input devices 108 (shown in FIGS. 1B and 1C). When apairing mode is initiated, the operator 104 operates the user inputsystem 106 to select a user input device to be associated with amanipulator. For example, the operator 104 selects one of the user inputdevices 108 a, 108 b. After the pairing mode is initiated, referring toFIG. 3A, the display device 107 presents the virtual selector 121 at aninitial location that does not satisfy an association condition toassociate the user input device with a manipulator. For example, in animplementation where the proximity (e.g., graphical proximity) of thevirtual selector 121 is an association condition, the initial locationis not proximate to the representations 122 a, 122 b of the instruments116 a, 116 b presented on the display device 107.

The operator 104 then operates the user input system 106 to control thelocation of the virtual selector 121 and reposition the virtual selector121 relative to the imagery. This repositioning of the virtual selector121 can be controlled by the operator 104 in a manner to select one ofthe representations 122 a, 122 b of the instruments 116 a, 116 b andhence select one of the manipulators 102 a, 102 b for association withthe selected user input device. In response to operation of the userinput system 106, the user input system 106 generates a set of signalsfor controlling a position of the virtual selector 121. In particular,the set of signals causes the display device 107 to reposition thevirtual selector 121. For example, the set of signals is processed bythe controller 110 to determine the virtual selector 121 should be movedrelative to the imagery; then the controller 110 causes the displaydevice 107 to reposition the virtual selector 121.

For selection of the manipulator 102 a to be associated with theselected user input device, the display device 107 is controlled in amanner such that the presented virtual selector 121 is repositionedrelative to the imagery to a location proximate a location of therepresentation 122 a. Based on the repositioning of the virtual selector121 relative to the represented location of the instrument 116 a, themanipulator 102 a supporting the instrument 116 a is associated with theuser input device. The controller 110 forms an association between themanipulator 102 a and the user input device in response to therepositioning of the virtual selector 121 relative to the representedlocation of the instrument 116 a satisfying an association condition.

In some implementations, input to the user input device can repositionbut cannot reorient the virtual selector 121. In other implementations,input to the user input device can reposition and reorient the virtualselector 121; the reorientation of the virtual selector 121 can beachieved using a similar technique as described for repositioning thevirtual selector 121 above.

In some implementations, an association condition used in associatingmanipulators to user input devices does not include the orientation ofthe virtual selector 121. In other implementations, an associationcondition used in associating manipulators to user input devicesincludes the orientation of the virtual selector 121.

As one example of an implementation with an orientation-relatedassociation condition, the association conditions include a firstcondition and a second condition. The first condition corresponds to thevirtual selector 121 being proximate to a location of a representation122 a, and the second condition corresponds to the virtual selector 121being oriented toward the representation 122 a (e.g. where the virtualselector 121 has an apex or a point as shown in FIG. 3A such that alongitudinal axis through the apex passes through the representation 122a).

Many of the following examples discussed focuses on changing theposition (e.g. location) of the virtual selector 121, and the positionor translational motion of the virtual selector 121 in comparison withlocation or motion based association conditions. However, similar to theexample of FIG. 3A, the other examples discussed in this disclosure mayalso be implemented with association conditions that include or do notinclude orientation considerations.

Referring to FIG. 3B, the association condition corresponds to thevirtual selector 121 overlapping with a region in the presented imagerydefined the represented location of the instrument 116 a. The user inputsystem 106 is operated such that the virtual selector 121 isrepositioned to a location proximate the represented location of theinstrument 116 a to satisfy this association condition. In someexamples, referring to FIG. 3B, the region defined by the representedlocation of the instrument 116 a corresponds to an area occupied by therepresentation 122 a in the presented imagery. Alternatively oradditionally, the region is defined by a represented location of adistal portion of the instrument 116 or a represented location of an endeffector of the instrument 116 a in the presented imagery. In an exampleassociation process, the virtual selector 121 is repositioned from itslocation shown in FIG. 3A to its location shown in FIG. 3B where thevirtual selector 121 overlaps with an area occupied by therepresentation 122 a. The overlap between the virtual selector 121 andthe area corresponds to the association condition to associate theinstrument 116 a with the user input device. In this regard, when thevirtual selector 121 is repositioned to achieve this overlap, theassociation condition is satisfied. When the association condition issatisfied, the controller 110 operates the display device 107 to presenta success indicator 133.

In some implementations, the controller 110 presents visual feedbackduring the association process. For example, in some implementations,the controller 110 presents overlaid with or proximate to therepresentation 122 a a color, number, text, graphic, or some othervisual indicator to indicate association statuses or proposesassociations. As specific examples, when the pairing mode is initiated,a green light or a flashing “O” near the representation 122 a canindicate that the instrument 116 a (and its manipulator 102 a) is in anunassociated state, and a red light or a steadily presented (notflashing) “O” can indicate that the instrument 116 a (and itsmanipulator 102 a) is in an associated state. In some cases, a yellowlight or a flashing or steadily presented “X” can provide a visualwarning.

In some implementations, the controller 110 causes visual feedbackindicating additional information about the statuses of the user inputdevices 108 with the manipulators 102. In some implementations, thevisual feedback indicates which user input device is recommended to beassociated with which manipulator, or indicates which user input devicewill be associated with which manipulator upon confirmation. As oneexample, where the user input device 108 a and the instrument 116 a (orits manipulator 102 a) is recommended to be associated with each other,or will be associated with which manipulator upon confirmation, both thevirtual selector 121 and the representation 122 a can flash matching,similar, or identical: color, number, text, graphics, flashing sequence,or other visual feedback.

In some implementations, the controller causes visual feedbackindicating which user input device has been, or is currently, associatedwith which instrument or manipulator. As one example, after the userinput device 108 a is associated with the instrument 116 a (or itsmanipulator 102 a) the controller 110 can steadily present a color,number, text, graphical pattern, or other visual feedback indicative ofthe association near the appropriate representation of the instrument.In various implementations, this steady presentation of color, number,text, graphics, or other textual feedback can last for the entireduration during which that user input device is associated with thatinstrument.

In the example shown in FIG. 3B, the controller 110 has caused anassociation indicator 128 b (shown as a letter “L”) to appear overlaidwith the representation 122 a to indicate the instrument 116 a'sassociation status with an input device identified by “L”. Theassociation indicator 128 b can be presented for part or the entireduration during which the user input device 108 a is associated with theinstrument116 a.

Other examples of association conditions are described with respect toFIGS. 7A-7E.

Referring to FIG. 4 and as described herein, an example of the system100 for performing an association process includes the manipulatorsystem 101, the controller 110, a user output system 202, and the userinput system 106. While described with respect to FIGS. 1, 2, 3A, and 3Bas including four manipulators, in some implementations, as shown inFIG. 4, the manipulator system 101 can include any number ofmanipulators. For example, the manipulator system 101 includes Nmanipulators (e.g., Manipulator 1 through Manipulator N, collectivelyreferred to as manipulators 102). Similarly, while described withrespect to FIGS. 1, 2, 3A, and 3B as including two user input devices,in some implementations, as shown in FIG. 4, the user input system 106includes any number of user input devices or user-operable portions ofthe user input system 106. For example, the user input system 106includes M user input devices (e.g., User Input Device 1 through UserInput Device M, collectively referred to as user input devices 108).Examples of the user input devices 108 include: joysticks, touchscreens,gloves, foot pedals, touchscreens, or handheld remotes.

In some implementations, the system 100 includes a sensor system 200.The sensor system 200 includes sensors operable to detect movement ofthe user input devices 108. The sensor system 200 can detect poses,e.g., positions, orientations, or both positions and orientations, ofthe user input devices 108 and the manipulators 102 in the environment10. Sensors of the sensor system 200 include, for example, infraredsensors, ultrasonic sensors, image capture devices, accelerometers,position encoders, optical sensors, or other appropriate sensors fordetecting motion and poses of the manipulators 102 and the user inputdevices 108.

The user output system 202 provides human-perceptible feedback to theoperator 104 and includes the display device 107. The feedback providedby the user output system 202 can include feedback provided during anassociation process or during the following mode to provide guidance tothe operator 104 for controlling the virtual selector 121 or forcontrolling the manipulators 102, respectively. Furthermore, the useroutput system 202 is operable to present the virtual selector 121 (e.g.,on the display device 107) during the pairing mode to enable theoperator 104 to select a user input device and a manipulator forassociating with one another. In some implementations, the user outputsystem 202 and the user input system 106 correspond to the console 103.

The system 100 can further include a memory storage element 204. Thememory storage element 204 can store data indicative of associationsformed between the manipulators 102 and the user input devices 108. Thecontroller 110 can retrieve these stored data to determine whether auser input device or a manipulator is in an associated state or anunassociated state. Referring to FIG. 5, the manipulators 102 and theuser input devices 108 are associated so that each user input device 108is associated with a distinct manipulator 102. As a result, the userinput devices 108 can be controlled by the operator 104 so that theassociated manipulators can be independently controlled. In some cases,each of the manipulators 102 is associated with a corresponding one ofthe user input devices 108. As a result, each of the manipulators 102can be controlled using the user input devices 108.

Example Processes

Referring to FIG. 6, a process 600 including an association process anda following process is presented with respect to the system 100described herein. The process 600 is performed by the user input system106 and the user output system 202, the manipulator system 101, thecontroller 110, other portions of the system 100 (e.g., the console 103,the system 113), or a combination of the foregoing. At operation 601, apairing mode is initiated to associate one or more user-operableportions of the user input system 106 with one or more manipulators ofthe manipulator system 101. At operation 602, the association process isperformed to associate a particular manipulator with a particularuser-operable portion of the user input system 106. The particularmanipulator and the particular user-operable portion can both beselected by the operator 104. Examples of further operations andsub-operations the operations 601 and 602 are described with respect toFIGS. 7A-7E, 8A, 8B, and 9.

At operation 603, a following mode is initiated so that, in a followingprocess, the manipulator can be controlled in response to operation ofthe user-operable portion. In some implementations, in the followingmode, the manipulator associated with the user-operable portion atoperation 602 can be moved in response to operation of the user-operableportion by the operator 104. In response to operation of theuser-operable portion, the user input system 106 generates a set of userinput signals for controlling a position of the manipulator. Thecontroller 110 then generates a corresponding set of control signalsbased on the set of user input signals. The set of control signals aretransmitted to the manipulator to move the manipulator with which theuser-operable portion is associated (e.g., during the pairing mode).This causes the manipulator and an instrument mounted to the manipulatorto move. In this regard, the user-operable portion and the manipulatorform a leader-follower system in which the user-operable portion is aleader device and the manipulator is a follower device, thereby enablingthe manipulator to be teleoperated through operation of theuser-operable portion. If the system 100 is a surgical system, aninstrument supported by the manipulator can be controlled to perform asurgical operation on a patient.

FIG. 7A illustrates an example of an association process 700 toassociate a particular user-operable portion of the user input system106 with a particular manipulator of the manipulator system 101. Theprocess 700 is performed, for example, during the operations 601 and 602described with respect to the process 600.

Operations 701-703 of FIG. 7A illustrate an example set operations forinitiating a pairing mode. At operation 701 of the process 700, theoperator 104 operates the user input system 106 to initiate the pairingmode. For example, the user input system 106 includes a user-operableportion dedicated to initialization of the pairing mode, and theoperator 104 operates the dedicated user-operable portion to initializethe pairing mode. This dedicated user-operable portion can correspond toa button that initiates the pairing mode when manually operated by theoperator 104. At operation 702, the user input system 106 transmits asignal to the controller 110 to initiate the pairing mode. At operation703, the controller 110 initiates the pairing mode.

Once in the pairing mode, a particular user-operable portion is selectedfor association. For example, the operator 104 operates the user inputsystem 106 to select a user-operable portion. Alternatively, thecontroller 110 automatically selects one of the user-operable portionsfor association. In the pairing mode, the operator 104 further providesan association intent to associate the particular user-operable portionwith a particular manipulator. In addition, feedback is provided to theoperator 104 so that the operator 104 can be kept informed of states ofthe manipulators of the manipulator system 101 and the user-operableportions of the user input system 106. Operations 704-713 illustrateexamples of operations that occur during the pairing mode.

In some implementations, after the pairing mode is initiated atoperation 703, at operation 704, the controller 110 transmits signals toprovide association indicators to the operator 104. The signals can betransmitted to the user output system 202. The user output system 202presents the association indicators to indicate association states ofeach of the manipulators of the manipulator system 101.

FIG. 7B illustrates an example of visual feedback that can be providedat the operation 703. The visual feedback includes associationindicators to provide information indicative of association states ofthe manipulators 102 a, 102 b (not shown). The association states of themanipulators 102 a, 102 b can be unassociated or associated, with anunassociated state indicating that a manipulator has not been associatedwith a user-operable portion and an associated state indicating that amanipulator has already been associated with a user-operable portion.

Referring to FIG. 7B, the display device 107 can present visual feedbackincluding a state indicator 132 a for the manipulator 102 a supportingthe instrument 116 a and a state indicator 132 b for the manipulator 102b supporting the instrument 116 b. The state indicators 132 a, 132 b arepositioned proximate distal portions of the representations 122 a, 122 bof the instruments 116 a, 116 b. The state indicator 132 a indicatesthat the manipulator 102 a is in an unassociated state, while the stateindicator 132 b indicates that the manipulator 102 b is in an associatedstate. The state indicators 132 a, 132 b can visually inform theoperator 104 of the association states of the manipulators 102 a, 102 bso that the operator 104 can provide association intent in view of theassociation states of the manipulators 102 a, 102 b.

Turning back to FIG. 7A, at operation 705, the operator 104 operates theuser input system 106 to provide an association intent. For example, theuser input system 106 generates the set of user input signals forcontrolling the position (e.g., location) and/or orientation of thevirtual selector 121. The set of user input signals is generated inresponse to the operation of the user input system 106. The followingdiscussion focuses on controlling the position of the virtual selector121. In implementations where the orientation of the virtual selector121 is also controlled, a similar process may be used to orient andreorient the virtual selector 121.

In some implementations, the operator 104 operates a user-operableportion of the user input system 106 to generate the set of signals. Theuser-operable portion that is operated can correspond to the particularuser-operable portion to be paired with a manipulator. In otherimplementations, the user input system 106 includes a user-operableportion dedicated for use by the operator 104 to cause repositioning ofthe virtual selector 121. In this regard, the user-operable portionoperated to control the position and the orientation of the virtualselector 121 can be different from the user-operable portions that canbe associated to the manipulators.

At operation 706, in response to the set of user input signals generatedby the user input system 106, the user output system 202 of the console103 repositions the virtual selector 121 (described with respect toFIGS. 3A and 3B) relative to the imagery presented by the user outputsystem 202. For example, the controller 110 generates the set of controlsignals in response to the set of user input signals and transmits theset of control signals to the user output system 202 to control theposition and orientation of the virtual selector 121.

The repositioning of the virtual selector 121 can occur in a number ofmanners. In some implementations, the virtual selector 121 is movablerelative to the imagery in response to the set of signals generated bythe user input system 106. The virtual selector 121 moves along acontinuous path from a first position to a second position in responseto the set of signals. For example, the user-operable portion includes auser input device such as a joystick, and the virtual selector 121 movesrelative to the imagery in response to manual manipulation of thejoystick. The user output system 202 presents the virtual selector 121on the display device 107 such that, when viewed by the operator 104,the virtual selector 121 appears to translate across the display device107. Similarly, the virtual selector 121 is movable through orientationsbetween a first orientation and a second orientation in response to theset of signals. In this regard, the virtual selector 121 appears torotate continuously.

In some implementations, rather than being moved across the displaydevice 107 relative to the imagery, the virtual selector 121 isrepositioned on the display device 107 from a first position to a secondposition on the display device 107 without moving along a path from thefirst position to the second position. Alternatively or additionally,the virtual selector 121 is repositioned on the display device 107 froma first orientation to a second orientation without continuouslyrotation from the first orientation to the second orientation. The userinput system 106 is operated to select a location or an orientation ofthe virtual selector 121 after repositioning, e.g., the second positionor the second orientation of the virtual selector 121. For example, theuser input system 106 can include a touchscreen, and the operator 104selects the location by touching a portion of the touchscreen. Inresponse to this selection, the display device 107 presents the virtualselector 121 at the second position or the second orientation absent anymovement of the virtual selector 121 between the first position and thesecond position or between the first orientation and the secondorientation.

Turning back to FIG. 7A, at operation 707, the controller 110 determineswhether the repositioning of the virtual selector 121 satisfies anassociation condition. Association conditions can vary betweenimplementations. Association conditions can include a condition for aposition of the virtual selector 121, a condition for an orientation ofthe virtual selector 121, or an amount of time that the virtual selector121 is at a particular position or within a region. Also, as discussedin connection with FIG. 3A and applicable to the various examplesdisclosed herein, association conditions can include a condition basedon an orientation of the virtual selector 121.

In some implementations, referring to FIG. 7C, the association conditionto associate the instrument 116 a with the user-operable portioncorresponds to the virtual selector 121 being repositioned to a locationon or proximate the representation 122 a of the instrument 116 a. Insome cases, the display device 107 presents selectable indicators 121 a,121 b proximate the representations 122 a, 122 b. In some cases, theselectable indicators 121 a, 121 b do not overlap with therepresentations 122 a, 122 b. The association condition is satisfiedwhen the virtual selector 121 is repositioned on or proximate one of theselectable indicators 121 a, 121 b. For example, the virtual selector121 overlaps the selectable indicator 121 a to satisfy the associationcondition for associating the user-operable portion with the manipulator102 a, or the virtual selector 121 overlaps the selectable indicator 121b to satisfy the association condition for the associating theuser-operable portion with the manipulator 102 b.

In some implementations, referring to FIG. 7D, the association conditionto associate the manipulator 102 a with the user-operable portioncorresponds to the virtual selector 121 being positioned within a region123 a surrounding the representation 122 a. For example, the region 123a includes a combination of (i) an area in the imagery covered by aportion 124 of the representation 122 a representing the end effector ofthe instrument 116 a and (ii) an area in the imagery surrounding theportion 124. The virtual selector 121 can thereby trigger associationwith the manipulator 102 a without overlapping with the area in theimagery covered by the representation 122 a.

In some implementations, the region 123 a is defined by a predefineddistance to a particular point on the representation 122 a. Theparticular point can be, for example, a centroid of the area covered bythe representation 122 a, a centroid of the area covered by the portion124 of the representation 122 a, or another point along therepresentation 122 a. Alternatively or additionally, the region 123 a isa shape that has a predefined size and that bounds the representation122 a or bounds the portion 124 of the representation 122 a. The shapeof the region 123 a can be, for example, rectangular, circular, ovular,or another appropriate shape.

The association condition can be satisfied immediately when the virtualselector 121 is repositioned into the region 123 a. In someimplementations, the controller 110 further requires that the virtualselector 121 is positioned within the region 123 a for a predefinedperiod of time, e.g., 0.5 seconds to 2 seconds, before the associationcondition is considered satisfied. In some implementations, thecontroller 110 further requires that the virtual selector 121 besubstantially stationary within the region 123 a for the predefinedperiod of time.

In some implementations, as the virtual selector 121 is beingrepositioned during the pairing mode, the controller 110 providesfeedback to the operator 104. The controller 110 provides the feedbackin response to repositioning of the virtual selector 121. For example,referring to FIG. 7D, the virtual selector 121 is repositioned into aregion 125 a proximate the representation 122 a. For example, the region125 a surrounds the representation 122 a as well as the region 123 a.The region 125 a thus encompasses at least the portion 124 of therepresentation 122 a.

In some implementations, the repositioning of the virtual selector 121that satisfies the association condition corresponds to movement of thevirtual selector 121 toward the representation 122 a of the instrument116 a. For example, the association condition is satisfied when avelocity or an acceleration of the virtual selector 121 is defined by avector that intersects the represented location of the instrument 116 aor the region 123 a.

FIG. 7E illustrates an example of visual feedback provided to theoperator 104 through the display device 107 after the virtual selector121 is repositioned to be within the region 125 a. In response to thevirtual selector 121 being repositioned into the region 125 a, thedisplay device 107 presents an information box 126, e.g., a tooltip,including information pertaining to the instrument 116 a and themanipulator 102 a, including a type 127 of the instrument 116 a and anassociation state 128 of the manipulator 102 a. The association state128 may include any appropriate amount of information regardingassociation status. In some implementations, the association state 128indicates just “associated” or “unassociated.” In some implementations,the association state 128 indicates with which input device theinstrument 116 a is associated. In the example shown in FIG. 7E, theassociation state 128 states that the state is “Associated” with InputDevice ‘L’“, and is supplemented with an association indicator 128 b(“L”) overlaid or proximate the representation 122 a.

The association state 128 and association indicator 128 b can beindicated by any one or combination of color, number, text, graphicalpattern, or other visual feedback. In some embodiments where the inputdevice 108 include visual feedback devices such as lights or displays,the input devices can also present matching, similar, or identicalcolors, numbers, text, graphics, or other visual feedback as the onesused for the representation of the associated instrument. In variousimplementations, the association indicator 128 b is presented for partor the entire duration during which the user input device 108 a isassociated with the instrument116 a.

As shown in the example of FIG. 7E, the instrument 116 a is a cutter,and the association state of the manipulator 102 a is an unassociatedstate. The display device 107 is also operated to provide an enlargedrepresentation 129 of the instrument 116 a as the virtual selector 121approaches the representation 122 a. The enlarged representation 129 canprovide visual confirmation to the operator 104 that the instrument 116a is the desired instrument for association with the user-operableportion. The operator 104 may be able to more easily identify theinstrument 116 a through the enlarged representation 129.

An enlarged representation 130 of the virtual selector 121 can also bepresented so that the operator 104 can monitor movement of the virtualselector 121 relative to the representation 122 a by monitoring movementof the enlarged representation 130 relative to the enlargedrepresentation 129. These enlarged representations 129, 130 can allowselection of the instrument 116 a to be easier by providing the operator104 with a larger target for selection using the virtual selector 121.

Turning back to FIG. 7D, in some implementations, the virtual selector121 is repositionable into a region 123 b for association with theinstrument 116 b. The region 123 b can have features similar to featuresof the region 123 a. Similarly, while repositioning of the virtualselector 121 into the region 125 a for the instrument 116 a is describedfor triggering feedback to be provided to the operator 104, in someimplementations, the virtual selector 121 is repositioned into a region125 b for triggering feedback to be provided. Movement of the virtualselector 121 can trigger provision of feedback related to the instrument116 b.

In some implementations, the controller 110 associates the manipulatorwith the user-operable portion only if the user-operable portion of theuser input system is in an unassociated state. Turning back to FIG. 7A,after the controller 110 determines that repositioning of the virtualselector 121 satisfies the association condition for a particularmanipulator, at operation 708, the controller 110 determines anassociation state of the manipulator. The controller 110 determineswhether the manipulator is in an unassociated state. For example, thecontroller 110 can access the memory storage element 204 (shown in FIG.4) to determine whether an association for the manipulator has beenstored on the memory storage element 204. If the manipulator is not inan unassociated state, e.g., is in an associated state, the operator 104at operation 709 either confirms that a new association is to beprovided to the manipulator or indicates that the manipulator shouldmaintain the stored association. If the operator 104 indicates that themanipulator should maintain the stored association, the operator 104operates the user input system at the operation 705 to provide anotherassociation intent to select another one of the manipulators.

If the operator 104 confirms that a new association is to be provided,the controller 110 can remove the stored association for themanipulator. If it is confirmed at the operation 709 that a newassociation is to be created for the manipulator or if it is determinedat the operation 708 that the manipulator is in an unassociated state,the controller 110 at operation 710 requests for user confirmation of anassociation between the user-operable portion and the manipulator. Forexample, the controller 110 transmits data representing the request forconfirmation to the user output system 202.

At operation 711, the operator 104 provides the confirmation of theassociation. In some implementations, the operator 104 can provide thisconfirmation by operating the user input system 106. For example, theoperator 104 can cause the virtual selector 121 to move to a predefinedregion presented on the display device 107 to confirm the association.The predefined region can correspond to a selectable button presented onthe display device 107.

At operation 712, after receiving confirmation of the association, thecontroller 110 stores the association, e.g., in the memory storageelement 204. The controller 110 then provides a success signal atoperation 713. For example, the user output system 202 is operated toprovide a human-perceptible signal indicative of the success of theassociation between the manipulator and the user input element. Thehuman-perceptible success signal can correspond to the success indicator133 described with respect to FIG. 3B.

While described with respect to associating a single user-operableportion with a single manipulator, in some implementations, operations704-713 can be repeated to associate other user-operable portions of theuser input system 106 with other manipulators of the manipulator system101. The system 100 can remain in the pairing mode until the operator104 operates the user input system 106 to provide input indicative ofinitiating the following mode, e.g., initiating operation 603. In thefollowing mode, the user-operable portions that have been associatedwith the manipulators can be operated by the operator 104 to controlmovement of the manipulators.

In some implementations, the controller 110 can provide recommendationsto optimize the associations formed between the manipulators and theuser-operable portions. Process 800 of FIG. 8A illustrates an exampleprocess to provide such a recommendation. The process 800 is initiatedafter the pairing mode is initiated. Upon initiation of the pairingmode, at operation 801, the user input system 106 transmits signalsindicative of poses of the user-operable portions of the user inputsystem 106, e.g., poses of the user-operable portions in the environment10 (shown in FIG. 1A). At operation 802, the manipulator system 101transmits signals indicative of poses of the manipulators of themanipulator system 101 to the controller 110. At operation 803, thecontroller 110 receives these signals from the user input system 106 andthe manipulator system 101. In some implementations, the sensor system200 detects the poses of the user-operable portions, the manipulators,or both and transmits these signals to the controller 110. In addition,the controller 110 further receives a signal indicative of the positionand the orientation of the image capture device, e.g., on the instrumentsupported on the manipulator 102 c.

The controller 110 receives the signals and uses kinematic modeling todetermine the positions and orientations of the manipulators 102 a, 102b, the positions and orientations of the instruments 116 a, 116 b, andthe position and orientation of the image capture device. In some cases,one or more signals are generated by sensors of the manipulators (e.g.,the manipulators 102 a, 102 b, and the manipulator to which the imagecapture device is mounted) or sensors of the instruments (e.g., theinstruments 116 a, 116 b, and the image capture device). The sensors ofthe manipulators include, for example, accelerometers, gyroscopes,encoders, or other sensors associated with joints of the manipulators102 a, 102 b. The sensors of the instruments include, for example, shapesensors through shafts of the instruments. Alternatively, the positionsand orientations of the manipulators and/or the positions andorientations of the instruments are determined based on one or moresignals from optical sensors (e.g., image capture devices). Themanipulators or the instruments are equipped with optical fiducialsdetectable by the optical sensors.

At operation 804, based on the received signals, the controller 110determines optimal associations between the manipulators of themanipulator system 101 and the user-operable portions of the user inputsystem 106. FIG. 8B diagrammatically depicts relative positions of thedisplay device 107 and the user input devices 108. In this example, theuser input devices 108 correspond to the user-operable portionsdescribed with respect to FIG. 8A. The representation 122 a of theinstrument 116 a appears on a left side of imagery presented on thedisplay device 107, while the representation 122 b of the instrument 116b appears on a right side of the imagery. To provide the operator 104with intuitive control of the instruments 116 a, 116 b as theinstruments 116 a, 116 b appear on the display device 107, thecontroller 110 provides a recommendation to associate the user inputdevice 108 a (in the left hand of the operator 104) with the instrument116 a represented on the left side of the imagery. Furthermore, thecontroller 110 provides a recommendation to associate the user inputdevice 108 b (in the right hand of the operator 104) with the instrument116 b represented on the right side of the imagery.

Turning back to FIG. 8A, the controller 110 can determine the relativepositions and orientations of the user-operable portions and themanipulators 102 a, 102 b based on the signals indicative of the posesof these devices. Alternatively, in some implementations, the controller110 determines the positions and orientations of the user-operableportions relative to the instruments 116 a, 116 b supported by themanipulators 102 a, 102 b. The controller 110 can determine relativeposes of the instruments 116 a, 116 b as they would appear to theoperator 104 on the display device 107. The controller 110 can determinea recommendation for the associations between the user-operable portionsand the manipulators 102 a, 102 b based on these relative poses. Invarious implementations, the recommendation may include recommendedassociations for a subset or all of the user input devices (e.g. 108 a,108 b) and a subset or all of the manipulators (102 a, 102 b). Also, invarious implementations, the recommendations may indicate degrees ofrecommendation for a particular association, such as: a more recommendedassociation between a user input device and a manipulator (e.g. betweenthe user input device 108 a and the manipulator holding the instrument116 a), a less recommended association between a user input device and amanipulator (e.g. between the user input device 108 a and a manipulatorholding an instrument not shown in the imagery), or a not recommendedassociation between a user input device and a manipulator (e.g. the userinput device 108 a and a manipulator holding the instrument 116 b).

In some implementations, the controller 110 does not receive positionsand orientations of the user-operable portions for determining therecommendations. The user-operable portions can be configured such thatthe user-operable portions have fixed positions and orientationsrelative to one another. In this regard, the controller 110 can providea recommendation based on the positions and orientations of themanipulators 102 a, 102 b relative to one another or based on thepositions and orientations of the instruments 116 a, 116 b relative toone another.

After the controller 110 determines the optimal associations, thecontroller 110 at operation 805 provides a signal to indicate theoptimal associations to the operator 104. For example, the controller110 controls the user output system 202 to provide an appropriate signalto guide the operator 104 to form the optimal associations between themanipulators 102 a, 102 b and the user-operable portions. For example,the display device 107 can be operated to present a recommendation forthe operator to form the optimal associations determined at theoperation 804. When a particular user-operable portion is selected forassociation, the controller 110 can cause the display device 107 topresent a recommendation to associate the particular user-operableportion with a recommended manipulator. Turning back to the example ofFIG. 8B, when the user input device 108 a is selected for association,the virtual selector 121 is presented on the display device 107, and thedisplay device 107 further provides a recommendation to reposition thevirtual selector 121 to select the manipulator 102 a to associate withthe user input device 108 a. Similarly, when the user input device 108 bis selected for association, the virtual selector 121 is presented onthe display device 107, and the display device 107 further provides arecommendation to reposition the virtual selector 121 to select themanipulator 102 b to associate with the user input device 108 b.

In some implementations, prior to initiating the following mode andafter the associations between user-operable portions and manipulatorsare formed, for each user-operable portion, poses of the user-operableportions and the manipulators can be adjusted to allow for easieroperation of the user-operable portions in the following mode. To ensurethat the operator 104 can control the manipulator 102 through its fullrange of motion using the user-operable portion, the portion of theuser-operable portion can be reoriented or repositioned. Process 900 ofFIG. 9 is performed to achieve this reorienting or repositioning of aportion of a user-operable portion.

At operation 901, the user input system 106 transmits a signalindicative of a pose of a portion of a user-operable portion of the userinput system 106. The signal can be indicative of the position andorientation of the user-operable portion relative to the full range ofmotion of the user-operable portion. For example, the user-operableportion can correspond to a joystick, and a position sensor of thesensor system 200 (shown in FIG. 4) that is coupled to the joystick cangenerate the signal.

At operation 902, the manipulator system 101 transmits a signalindicative of a pose of a manipulator of the manipulator system 101.Position sensors of the sensor system 200, e.g., encoders,accelerometers, etc., can generate and transmit the signal. At operation903, the controller 110 receives these signals from the user inputsystem 106 and the manipulator system 101.

Based on these signals, at operation 904, the controller 110 determineswhether a pose of the user-operable portion relative to a full range ofmotion of the user-operable portion matches with a pose of themanipulator relative to a full range of motion of the manipulator. Forexample, the user-operable portion can have a degree of freedom ofmotion for controlling yaw motion of the distal end of the manipulator.The controller 110 determines whether the position of the user-operableportion within the full range of motion for this degree of freedom ofmotion matches with the position of the distal end of the manipulatorwithin the full range of motion for its yaw degree of freedom. Thecontroller 110 similarly compares the position of the portion of theuser-operable portion for each of its other degrees of freedom to theposition of the manipulator for its other degrees of freedom.

If the poses of the portion of the user-operable portion and themanipulator do not match, the controller 110 at operation 905 transmitssignals to reorient the portion of the user-operable portion. Atoperation 906, the user input system 106 receives the signals. In somecases, the signals cause automatic motion of the portion of theuser-operable portion. For example, the signals drive one or moreactuators to move the portion of the user-operable portion.Alternatively, the user input system 106 provides feedback to theoperator 104 to reorient or reposition the portion of the user-operableportion. The user input system 106 then at operation 901 transmitsanother signal indicative of the pose of the portion of theuser-operable portion, and the controller 110 determines again whetherthere is match between the poses of the portion of the user-operableportion and the manipulator.

When the controller 110 determines a match at operation 904, thefollowing mode can be initiated. For example, the success signal can beprovided at operation 713 of the process 700, and the following mode canthen be initiated.

Further Implementations

A number of implementations have been described, and it is contemplatedthat various combinations, additions, or modifications may be made. Forexample, the above implementations may be combined in any appropriatemanner. As another example, an implementation may include none, one, ora plurality of any of the following.

For example, controllers, processors, and any associated componentsdescribed herein can be part of a computing system that facilitatescontrol of the systems according to processes and methods describedherein. FIG. 10 is a schematic diagram of an example of a computersystem 1000 that can be used to implement a controller, e.g., thecontroller 110 or other controller of the system 100, described inassociation with any of the computer-implemented methods describedherein, e.g., methods including one or more of the processes oroperations described with respect to FIGS. 6-9. The system 1000 includescomponents such as a processor 1010, a memory 1020, a storage device1030, and an input/output device 1040. The components 1010, 1020, 1030,and 1040 are interconnected using a system bus 1050. The processor 1010is capable of processing instructions for execution within the system1000. In some examples, the processor 1010 is a single-threadedprocessor, while in some cases, the processor 1010 is a multi-threadedprocessor. The processor 1010 is capable of processing instructionsstored in the memory 1020 or on the storage device 1030 to displaygraphical information for a user interface on the input/output device1040.

Memory storage for the system 1000 can include the memory 1020 as wellas the storage device 1030. The memory 1020 stores information withinthe system 1000. The information can be used by the processor 1010 inperforming processes and methods described herein. In some examples, thememory 1020 is a computer-readable storage medium. The memory 1020 caninclude volatile memory and/or non-volatile memory. The storage device1030 is capable of providing mass storage for the system 1000. Ingeneral, the storage device 1030 can include any non-transitory tangiblemedia configured to store computer readable instructions. Optionally,the storage device 1030 is a computer-readable medium. Alternatively,the storage device 1030 may be a floppy disk device, a hard disk device,an optical disk device, or a tape device.

In some cases, the processor 1010 is in communication with a remotecomputing system 1035. The remote computing system 1035 includes, forexample, a remote server, a cloud computing device, or other computingdevice remote from the processor 1010 and its systems. The remotecomputing system 1035 includes computing resources remote from theenvironment of the processor 1010, e.g., remote from the surgicalenvironment. In some cases, the remote computing system 1035 includesone or more servers that establish wireless links with the processor1010. The remote computing system 1035 includes, for example, a portionof a network-accessible computing platform implemented as a computinginfrastructure of processors, storage, software, data access, and soforth accessible by the processor 1010.

The system 1000 includes the input/output device 1040. The input/outputdevice 1040 provides input/output operations for the system 1000. Insome examples, the input/output device 1040 includes a keyboard, acomputer mouse, a pointing device, a voice-activated device, amicrophone, a touchscreen, etc. In some cases, the input/output device1040 includes a display unit for displaying graphical user interfaces.

The features of the methods and systems described in this applicationcan be implemented in digital electronic circuitry, or in computerhardware, firmware, or in combinations of them. The features can beimplemented in a computer program product tangibly stored in aninformation carrier. The information carrier can be, for example, amachine-readable storage device, for execution by a programmableprocessor. Operations, e.g., of the processes 600, 700, 800, and 900,can be performed by a programmable processor executing a program ofinstructions to perform the functions described herein by operating oninput data and generating output. The described features can beimplemented in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program includes a set ofinstructions that can be used, directly or indirectly, in a computer toperform a certain activity or bring about a certain result. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages. The computer program can be deployedin any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. The computer program implements, for example, a fastgenetic algorithm (FGA).

Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles. Such devices can include magnetic disks, such as internal harddisks and removable disks, magneto-optical disks, and optical disks.Storage devices suitable for storing the computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices, magnetic disks such as internal hard disks and removabledisks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (Cathode Ray Tube)or LCD (Liquid Crystal Display) or OLED (Organic Light Emitting Diodes)monitor for displaying information to the user and one or more inputdevices by which the user can provide input to the computer, suchkeyboards, buttons, switches, pedals, computer mice, touchpads, touchscreens, joysticks, or trackballs. Alternatively, the computer can haveno keyboard, mouse, or monitor attached and can be controlled remotelyby another computer. In some implementations, the display deviceincludes a head mounted display device or an augmented reality displaydevice (e.g., augmented reality glasses).

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include, e.g., a LAN, a WAN, and thecomputers and networks forming the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork. The relationship of client and server arises by virtue ofcomputer programs running on the respective computers and having aclient-server relationship to each other.

The processor 1010 carries out instructions related to a computerprogram. The processor 1010 can include hardware such as logic gates,adders, multipliers and counters. The processor 1010 can further includea separate arithmetic logic unit (ALU) that performs arithmetic andlogical operations.

The association process 700 is described as being performed to associatea particular user-operable portion of the user input system 106 with aparticular manipulator. In some implementations, the user-operableportion corresponds to a user input device of the user input system 106,such as a joystick. During the process 700, user input devices of theuser input system 106 are each associated with a correspondingmanipulator of the manipulator system 102. In some implementations,rather than associating a particular user input device with a particularmanipulator, a particular user-operable portion of the user input system106 is associated with a particular manipulator during the associationprocess 700. For example, the user input system 106 can include a userinput device having multiple distinct user-operable portions. If theuser input device is a touchscreen device, the distinct user-operableportions correspond to user interface elements positioned on differentportions of the touchscreen device. In this regard, each user interfaceelement can be associated with a corresponding manipulator during theassociation process 700.

While the manipulators 102 are described and shown as being distinctmanipulators separately mounted with mounting locations movable relativeto each other, e.g., to an operating table, the association processesdescribed herein are also applicable to manipulators that are mounted toa shared base. For example, referring to FIG. 11, a manipulator system1101 includes manipulators 1102 a, 1102 b, 1102 c, 1102 d (collectivelyreferred to as manipulators 1102), each of which is mounted to a commonbase 1104. A joint 1106 can be driven to reorient all of themanipulators 1102. The base 1104 can be mounted to a movable cartportion 1108. The movable cart portion 1108 is, for example, supportedabove a floor surface by wheels. In this regard, the manipulator system1101 is easily movable about an environment.

While a single user-operable portion is described as being associatedwith a single manipulator during the process 700, in someimplementations, a set of user-operable portions is associated to amanipulator during an association process. For example, both a left orright joystick and a corresponding left or right foot pedal may beassociated to the same manipulator. This enables the operator to controldifferent features of the manipulator, e.g., a position, a velocity,etc., of the manipulator using multiple user-operable portions. In someimplementations, instead of having to individually associate eachuser-operable portion in the set of user-operable portions, the operatorcan simultaneously associate each user-operable portion of the set ofuser-operable portions to the manipulator. For example, if the set ofuser-operable portions includes both a pedal and a joystick, theoperator provides the association intent at the operation 705 to movethe virtual selector 121 and initiate association of both the pedal andthe joystick together to the manipulator.

While the display device 107 shown in FIG. is described as providingvisual feedback to the operator 104, in some implementations, otherindicator devices are operated to provide a human-perceptible indicationindicative of information pertaining to an instrument or pertaining toprogress of an association process. For example, indicator devices canprovide human-perceptible tactile feedback, aural feedback, or acombination thereof. If the indicator devices provide tactile feedback,the tactile feedback can include vibro-tactile feedback, force feedback,or other forms of feedback associated with a user's sense of touch. Theindicator devices can include, for example, a vibration generator. Forexample, in implementations in which the user input system 106 ismanually operable, the indicator devices can be coupled to the userinput system 106 and can generate vibrations that serve as hapticfeedback for the operator 104 when the operator 104 is manuallyoperating the user input system. If the indicator devices provide auralfeedback, the indicator devices include, for example, an audio outputdevice such a speaker. In such cases, the indicator devices can narrateaudible feedback to the operator.

While the region 123 a shown in FIG. 7B is described as surrounding theportion 124 of the representation 122 a of the instrument 116 a(corresponding to the end effector of the instrument 116 a), in someimplementations, the region 123 a surround another portion of therepresentation. In some examples, the region 123 a surrounds a portionof the representation 122 a corresponding to a particular component ofthe instrument 116 a, such as a pivot of the end effector, a joint ofthe end effector, a shaft of the instrument 116 a, or other portion ofthe instrument 116 a.

The pairing mode can be initiated in response to a particular event. Forexample, operations 701-703 illustrate a particular example ofinitiating the pairing mode in response to operation of the user inputsystem 106. In some implementations, the user input device of the userinput system 106 that is operated to initiate the pairing modecorresponds to a user input device operable to initiate a clutching modein which the manipulators can be manually repositioned. In the clutchingmode, brake systems of the manipulators are disabled or joints of themanipulators are released so that the manipulators can be manuallyrepositioned by the operator. In some examples, the pairing mode is alsoinitiated when the clutching mode is initiated.

In some implementations, the pairing mode can be initiated in responseto events that are not associated with operation of manually operableuser input devices. For example, the controller 110 can be configured toinitiate the pairing mode when the system 100 is initialized. In somecases, the system 100 includes an audio input system that detects voicecommands issued by the operator 104. The operator 104 can utter a voicecommand, and the controller 110 accordingly initiates the pairing mode.Alternatively or additionally, the pairing mode can be initiated when anew operator accesses and operates the user input system 106.

In some implementations, in the pairing mode, the user-operable portionto be associated in the process 700 is selected by the operator 104 asdescribed herein. In other implementations, the user-operable portion isselected by the controller 110. For example, the controller 110 selectsa particular user-operable portion that is in an unassociated state. Ifthe user input system 106 includes multiple user-operable portions inunassociated states, the controller 110 selects a user-operable portionbased on relative association priorities of the user-operable portions.For example, the controller 110 by default can start with selecting aleftmost user-operable portion and sequentially select user-operableportions to the right of the leftmost user-operable portion. Thisselection scheme can be intuitive for the operator 104 and can reducethe number of operator steps required during the process 700.

While the visual feedback shown in FIG. 7B and provided at the operation703 is described as being indicative of association states of themanipulators, in some implementations, feedback provided at operation703 is indicative of association states of the user-operable portions.For example, the display device 107 presents graphic indicatorsindicative of available user-operable portions for association andfurther presents association state indicators to indicate theassociation states of the available user-operable portions.

While the processes have been described as being used for association ofthe user-operable portions with the manipulators 102, in someimplementations, the system 100 includes one or more sensors that detectmotion and form an association based on the detected motion. Forexample, the processes described herein are used for association ofhands of the operator 104. Referring to FIG. 12, in someimplementations, the system 100 includes an optical motion detectionsystem 1200 including optical sensors 1202 a, 1202 b. The optical motiondetection system 1200 is part of the user input system 106 and isoperable to move the virtual selector 121 described herein. Rather thanincluding a console 103 with a user output system, the system 100includes a user output system including a standalone display device 1206for presenting imagery of the instruments and presenting imagery of thevirtual selector 121. In this regard, the display device 1206 is similarto the display device 107 described herein.

The optical sensors 1202 a, 1202 b can provide a stereoscopic imagery ofthe operator 104 and can be used to detect motion of the operator 104,in particular, motion of hands 1204 a, 1204 b of the operator 104.Movements of the hands 1204 a, 1204 b can be used to control movement ofthe manipulators 102 in the following mode. For example, the hands 1204a, 1204 b are moved in a pattern or sequence in accordance to predefinedgestures for controlling the system 100. The predefined gestures caninclude a gesture for initiating a pairing mode, a gesture for proposingan association between a hand and a manipulator, a gesture forinitiating a following mode, or other appropriate gesture to control thesystem 100. In some implementations, the hands 1204 a, 1204 b areequipped with gloves detectable by the optical motion detection system1200.

In addition, in accordance to the association processes describedherein, the operator 104 moves hands 1204 a, 1204 b to control thelocation of the virtual selector 121. The optical motion detectionsystem 1200 detects the movements of one of the hands 1204 a, 1204 b,and the controller 110 controls the location of the virtual selector 121based on the movements of the hand. When the hands 1204 a, 1204 b aremoved in a manner to satisfy the association conditions, the controller110 forms the associations between the hands 1204 a, 1204 b and thecorresponding manipulators. For example, at operation 705, the operator104 moves a hand 1204 a or a hand 1204 b to control the virtual selector121 to satisfy the association condition. The hands 1204 a, 1204 b canthen be used in the following mode to control motion of themanipulators.

The user input system 106 is described as including, in someimplementations, a user-operable portion for controlling the position ofthe virtual selector 121 that is distinct from the user-operableportions that can be associated to the manipulators. In some examples,the user-operable portion for controlling the position of the virtualselector 121 includes a joystick, a touchscreen, or another manuallyoperable user input device.

In some examples, referring to FIG. 13, the user-operable portion of theuser input system 106 includes an eye tracking system 1300 that detectsmotion of a gaze of the operator 104 as the operator 104 views thedisplay device 107. In the example shown in FIG. 13, the eye trackingsystem 1300 is part of the console 103 and detects motion of the eyes ofthe operator 104 when the eyes are placed on eyepieces 1302 of theconsole 103. In other examples, the eye tracking system 1300 is part ofa non-console input system such as the system 113. During operation, theoperator 104 can operate the eye tracking system 1300 by shifting thegaze of the operator 104. When the eyes are positioned on the eyepiecesto view the display device 107, the eye tracking system 1300 detectsmotion of the gaze of the eyes and generates one or more signalsindicative of the movement of the gaze. The one or more signals cancorrespond to the set of signals for controlling the position of thevirtual selector 121. The display device 107 can then be operated by thecontroller 110 based on the set of signals to cause the virtual selector121 to be moved or repositioned relative to the presented imagery on thedisplay device 107.

As described herein, a manipulator is associated with a user-operableportion so that the manipulator is movable in response to certainoperations of the user-operable portion. Thus, in some implementations,the associated user-operable portion can be used for controllingmovement of the manipulator. Further, in some implementations, theassociated user-operable portion is operable to control other functionsof the manipulator or an instrument mounted to the manipulator insteadof, or in addition to, controlling movement of the manipulator. In thisregard, at operation 603, when the following mode is initiated, themanipulator is not necessarily moved in response to operation of theassociated user-operable portion but, rather, receives a signal toperform a particular function or cause the instrument to perform aparticular function. For example, in some implementations where theinstrument is an image capture device, the associated user-operableportion is operable to control an image capture function of the imagecapture device, such as a zoom setting, a lighting setting, a shutterspeed setting, or other image capture setting. As another example, insome implementations where the instrument is a suction or irrigationdevice, the associated user-operable portion is operable to control theapplication of suction or irrigation. In some implementations where theinstrument is an image capture device, the associated user input deviceis operable to control the image capture device to capture imagery. Insome implementations where the instrument is a cauterizing device orother energy application device, the associated user input device isoperable to control the energy application device to apply energy totissue.

In some implementations, in the pairing mode, multiple manipulators areassociated with a single user-operable portion of the user input system106. For example, two or more manipulators can be associated with asingle one of the user-operable portions 108. When a singleuser-operable portion is associated with multiple manipulators, theuser-operable portion is operable to generate movement of each of themanipulators. For example, in some implementations, if the operator 104wishes to shift the combined workspace of multiple manipulators or theirassociated instruments to a different workspace, the operator 104 canoperate the user-operable portion to shift each of the manipulators to avicinity of this different workspace. In some implementations, ratherthan moving each of the manipulators one-by-one to reach the differentworkspace, the operator 104 can associate all of the manipulators to bemoved, with a single user-operable portion and operate the singleuser-operable portion to move the plurality of manipulators, as a group,to the vicinity of the different workspace.

As another example, in some implementations, multiple manipulators canbe associated with a single user-operable portion of the user-operableportions, and the single user-operable portion controls only one of themanipulators at a time. In some implementations, an operator selectswhich one of the manipulators is to be controlled by operating thesingle user-operable portion via an appropriate method, such asdepression of a button, turning of a dial, clicking of a pedal, voicecommands, etc. In some implementations, the operator operates a buttonor pedal is used to cycle through manipulators of the manipulators untilthe one to be controlled becomes active.

Alternatively or additionally, two or more user-operable portions can beassociated with a single manipulator. For example, in someimplementations, one of the user-operable portions associated with themanipulator is operable to move the manipulator, while the other of theuser-operable portions associated with the manipulator is operable tocontrol a non-movement function of the manipulator or a function of aninstrument mounted to the manipulator. In some implementations, the twoor more associated user-operable portions each is operable to control adifferent degree of freedom or a different set of degrees of freedom ofthe manipulator. For example, in some implementations, one of theuser-operable portions is manually operable to control a pitch, a yaw,and a roll motion of the manipulator, while the other of theuser-operable portions is manually operable to control movement of theinstrument relative to the manipulator along the insertion axis or tocontrol actuation of an end effector of the instrument. As yet anotherexample, in some implementations, a plurality of user-operable portionsare used to enable a multi-handed input. For example, positions,separation distances, direction of motion, speed of motion of theuser-operable portions, relative to each other or a reference, can beused to control the manipulator or an instrument supported by themanipulator.

As a specific example, in an implementation, two user input devices areassociated with a single manipulator holding an imaging system such as acamera. An operator holding a user input devices in each hand cancontrol the imaging system with two-handed combination input thatsimulates manipulation of the work piece relative to the imaging system.For example, in a camera implementation, combined motion of both inputdevices away from the operator moves the camera away or causes thecamera to zoom out, as if the work piece has been pushed away. Asanother example, in a camera implementation, combined motion of bothinput devices around a common center rotates the camera field of view,as if the work piece had been rotated. As a further example, in a cameraimplementation, an increase in the separation distance between the userinput devices causes the camera to zoom out, and a decrease in theseparation distance between the user input devices causes the camera tozoom in.

In some implementations, the controller 110 is configured todisassociate one or more user input device 108 from one or moremanipulators 102 in response to user input or a system event. As anexample, the controller 110 may be configured to disassociate anassociated pair of manipulator and user input device in response toreceiving a signal indicative of a user request to disassociate thefirst manipulator and the user input device. As another example, thecontroller may be configured to disassociate all manipulators associatedwith a user input device, or all user input devices associated with amanipulator, in response to receiving a signal indicative of a userrequest to disassociate such user input device or such manipulator. Insome implementations, the user input system 106 includes disassociatinguser-operable portions for initiating disassociation of theuser-operable portions from the manipulators 102. For example, each ofthe user input device 108 may comprise disassociating controls orfeatures. As another example, for each of the user-operable portions, acorresponding one of the disassociating user-operable portions can beoperated to disassociate a user-operable portion from a manipulator. Inaddition, in some cases, operation of a disassociating user-operableportion can also initiate the pairing mode.

Accordingly, other implementations are within the scope of the claims.

1-52. (canceled)
 53. A computer-assisted medical system comprising: afirst manipulator; a user input system; a user output system comprisinga display device; and a controller configured to execute instructions toperform operations, the operations comprising: in a pairing mode and inresponse to a first set of signals generated by the user input system,causing a virtual selector shown on the display device to move relativeto imagery shown on the display device, wherein the imagery represents alocation of a first component of the first manipulator, in the pairingmode, associating the first manipulator with a portion of the user inputsystem based on movement of the virtual selector relative to therepresented location of the first component, and in a following mode,controlling motion of the first manipulator in accordance with a secondset of signals, the second set of signals generated by the user inputsystem in response to user operation of the portion of the user inputsystem.
 54. The computer-assisted medical system of claim 53, furthercomprising a second manipulator, wherein the imagery further representsa location of a second component of the second manipulator, and whereinthe operations further comprise: in the pairing mode, associating thesecond manipulator with a portion of the user input system based onmovement of the virtual selector relative to the represented location ofthe second component; and in a following mode, after the secondmanipulator has been associated with the portion of the user inputsystem, controlling motion of the second manipulator in accordance witha third set of signals, the third set of signals generated by the userinput system in response to further operation of the portion of the userinput system associated with the second manipulator.
 55. Thecomputer-assisted medical system of claim 53, wherein the operationsfurther comprise: initiating the pairing mode when the computer-assistedmedical system is initialized; or initiating the pairing mode inresponse to receiving a signal from the user input system indicative ofa request to initiate the pairing mode.
 56. The computer-assistedmedical system of claim 53, wherein the operations further comprise:providing feedback indicative of a proposed association between theportion of the user input system and the first manipulator based on themovement of the virtual selector; and associating the portion of theuser input system with the first component in response to receiving asignal indicative of a user confirmation of the proposed association.57. The computer-assisted medical system of claim 53, wherein theoperations further comprise: causing the user output system to generatea human-perceptible indication of an association state of the firstmanipulator, the human-perceptible indication including visual feedback,aural feedback, or tactile feedback.
 58. The computer-assisted medicalsystem of claim 53, wherein associating the first manipulator with theportion of the user input system comprises: associating the firstmanipulator with the portion of the user input system in response to thevirtual selector moving to a first selectable indicator presented by thedisplay device; or associating the first manipulator with the portion ofthe user input system in response to the virtual selector overlappingwith a region in the imagery defined by the represented location of thefirst component.
 59. The computer-assisted medical system of claim 53,wherein associating the first manipulator with the portion of the userinput system comprises: associating the first manipulator with theportion of the user input system in response to the virtual selectorbeing within a predefined distance from the represented location of thefirst component; or associating the first manipulator with the portionof the user input system in response to the virtual selector movingtoward the represented location of the first component.
 60. Thecomputer-assisted medical system of claim 53, wherein: the user inputsystem comprises a plurality of user input devices; a first user inputdevice of the plurality of user input devices being operable to generatethe first set of signals; and the portion of the user input systemcomprises the first user input device.
 61. The computer-assisted medicalsystem of claim 53, wherein the operations further comprise: initiatingthe following mode after an orientation of the portion of the user inputsystem is aligned with an orientation of a representation of the firstcomponent in the imagery.
 62. The computer-assisted medical system ofclaim 61, wherein the operations further comprise: generating motion ofthe portion of the user input system to align the orientation of theportion of the user input system with the orientation of therepresentation of the first component in the imagery; or guiding manualpositioning of the portion of the user input system to align theorientation of the portion of the user input system with respect to theorientation of the representation of the first component in the imagery.63. The computer-assisted medical system of claim 53, wherein: thecomputer-assisted medical system further comprises a second manipulator;the imagery further represents a location of a second component of thesecond manipulator; and the operations further comprise: guidingassociation of the first manipulator with the portion of the user inputsystem based on positions or orientations of representations of thefirst component and the second component in the imagery.
 64. Thecomputer-assisted medical system of claim 53, wherein associating thefirst manipulator with the portion of the user input system comprises:associating the first manipulator with the portion of the user inputsystem only if another manipulator of the computer-assisted medicalsystem is not associated with the portion of the user input system; orassociating the first manipulator with the portion of the user inputsystem only if the portion of the user input system is in anunassociated state; or associating the first manipulator with theportion of the user input system only if the first manipulator is in anunassociated state.
 65. The computer-assisted medical system of claim53, wherein the operations further comprise: disassociating the firstmanipulator with the portion of the user input system in response toreceiving a signal indicative of a user request to disassociate thefirst manipulator or the portion of the user input system.
 66. Thecomputer-assisted medical system of claim 53, further comprising asecond manipulator, wherein the user input system comprises a pluralityof user input devices, wherein the portion of the user input systemcomprises a first user input device of the plurality of user inputdevices, and wherein the operations further comprise: in the pairingmode and in response to a third set of signals generated by a seconduser input device of the plurality of user input devices, causing asecond virtual selector shown on the display device to move relative tothe imagery shown on the display device, wherein the imagery furtherrepresents a location of a second component of the second manipulator,in the pairing mode, associating the second manipulator with the seconduser input device based on movement of the second virtual selectorrelative to the represented location of the second component, and in thefollowing mode, controlling motion of the second manipulator inaccordance with a fourth set of signals, the fourth set of signalsgenerated by the user input system in response to user operation of theportion of the user input system.
 67. A method of operating acomputer-assisted medical system comprising a first manipulator, themethod comprising; causing a display device to present imagery and avirtual selector, the imagery representing a location of a firstcomponent of the first manipulator; causing the display device to rendermovement of the virtual selector relative to the imagery in response toa first set of signals generated by a user input system; associating, ina pairing mode, the first manipulator with a portion of the user inputsystem based on movement of the virtual selector relative to therepresented location of the first component; and controlling, in afollowing mode, motion of the first component in accordance with asecond set of signals generated by the user input system in response touser operation of the portion of the user input system.
 68. The methodof claim 67, wherein the imagery further represents a location of asecond component of a second manipulator of the computer-assistedmedical system, and wherein the method further comprises: in the pairingmode, associating the second manipulator with a portion of the userinput system based on movement of the virtual selector relative to therepresented location of the second component; and in a following mode,after the second manipulator has been associated with the portion of theuser input system, controlling motion of the second manipulator inaccordance with a third set of signals, the third set of signalsgenerated by the user input system in response to further operation ofthe portion of the user input system associated with the secondmanipulator.
 69. The method of claim 67, further comprising: providingfeedback indicative of a proposed association between the portion of theuser input system and the first manipulator based on the movement of thevirtual selector; and associating the portion of the user input systemwith the first component in response to receiving a signal indicative ofa user confirmation of the proposed association.
 70. The method of claim67, wherein associating the first manipulator with the portion of theuser input system comprise: associating the first manipulator with theportion of the user input system in response to the virtual selectormoving to a first selectable indicator presented by the display device;or associating the first manipulator with the portion of the user inputsystem in response to the virtual selector overlapping with a region inthe presented imagery defined the represented location of the firstcomponent; or associating the first manipulator with the portion of theuser input system in response to the virtual selector being within apredefined distance from the represented location of the firstcomponent; or associating the first manipulator with the portion of theuser input system in response to the virtual selector moving toward therepresented location of the first component.
 71. The method of claim 67,wherein: the imagery further represents a location of a second componentassociated with a second manipulator of the computer-assisted medicalsystem; and the method further comprises: guiding association of thefirst manipulator with the portion of the user input system based onpositions or orientations of representations of the first and secondcomponents in the presented imagery.
 72. One or more non-transitorycomputer readable media storing instructions that are executable by aprocessing device, and upon such execution cause the processing deviceto perform operations comprising: causing a display device to presentimagery and a virtual selector, the imagery representing a location of afirst component associated with a first manipulator of acomputer-assisted medical system; causing the display device to rendermovement of the virtual selector relative to the imagery in response toa first set of signals generated by a user input system; associating, ina pairing mode, the first manipulator with a portion of the user inputsystem based on movement of the virtual selector relative to therepresented location of the first component; and controlling, in afollowing mode, motion of the first component in accordance with asecond set of signals generated by the user input system in response touser operation of the portion of the user input system.
 73. The one ormore non-transitory computer readable media of claim 72, wherein theimagery further represents a location of a second component associatedwith a second manipulator of the computer-assisted medical system, andwherein the operations further comprise: guiding association of thefirst manipulator with the portion of the user input system based onpositions or orientations of representations of the first and secondcomponents in the presented imagery.